An Information Theoretic Interpretation of Variational Inference based on the MDL Principle and the Bits-Back Coding Scheme

As we will see during this talk, the Bayesian and information-theoretic views of variational inference provide complementary and mutually beneficial perspectives to the same problems with two different languages. More specifically, based on the paper by Honkela and Valpola [4], we will provide an interpretation of variational inference based on the MDL principle as a theoretical framework for model evaluation using code length and on Bits-back as a practical efficient coding scheme which builds on the MDL principle. From this perspective, we can better understand the shortcomings of practical variational inference and the over-pruning phenomenon of variational auto-encoders.